{
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "6RLNvyXlDhG2"
      },
      "source": [
        "[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/pinecone-io/examples/blob/master/learn/generation/better-rag/00-rerankers.ipynb) [![Open nbviewer](https://raw.githubusercontent.com/pinecone-io/examples/master/assets/nbviewer-shield.svg)](https://nbviewer.org/github/pinecone-io/examples/blob/master/learn/generation/better-rag/00-rerankers.ipynb)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "FZ6sj8gPDhG4"
      },
      "source": [
        "# Rerankers"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "h8o5TRVfDhG4"
      },
      "source": [
        "Rerankers have been a common component of retrieval pipelines for many years. They allow us to add a final \"reranking\" step to our retrieval pipelines \u2014 like with **R**etrieval **A**ugmented **G**eneration (RAG) \u2014 that can be used to dramatically optimize our retrieval pipelines and improve their accuracy.\n",
        "\n",
        "In the example notebook we'll learn how to create retrieval pipelines with reranking using the [Cohere reranking model](https://txt.cohere.com/rerank/) (which is available for free).\n",
        "\n",
        "To begin, we setup our prerequisite libraries."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 1,
      "metadata": {
        "id": "thtg9njP4bOh"
      },
      "outputs": [],
      "source": [
        "!pip install -qU \\\n",
        "    datasets==2.14.5 \\\n",
        "    openai==1.6.1 \\\n",
        "    pinecone-client==3.1.0 \\\n",
        "    cohere==4.27"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "VReBq2IeDhG5"
      },
      "source": [
        "## Data Preparation"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "eY3OglQm4bOj"
      },
      "source": [
        "We start by downloading a dataset that we will encode and store. The dataset [`jamescalam/ai-arxiv-chunked`](https://huggingface.co/datasets/jamescalam/ai-arxiv-chunked) contains scraped data from many popular ArXiv papers centred around LLMs. Including papers from Llama 2, GPTQ, and the GPT-4 technical paper."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 2,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "pQAVgquj4bOk",
        "outputId": "2d76cbce-bad3-41aa-9f44-2c15d232ac47"
      },
      "outputs": [
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "/usr/local/lib/python3.10/dist-packages/huggingface_hub/utils/_token.py:72: UserWarning: \n",
            "The secret `HF_TOKEN` does not exist in your Colab secrets.\n",
            "To authenticate with the Hugging Face Hub, create a token in your settings tab (https://huggingface.co/settings/tokens), set it as secret in your Google Colab and restart your session.\n",
            "You will be able to reuse this secret in all of your notebooks.\n",
            "Please note that authentication is recommended but still optional to access public models or datasets.\n",
            "  warnings.warn(\n"
          ]
        },
        {
          "data": {
            "text/plain": [
              "Dataset({\n",
              "    features: ['doi', 'chunk-id', 'chunk', 'id', 'title', 'summary', 'source', 'authors', 'categories', 'comment', 'journal_ref', 'primary_category', 'published', 'updated', 'references'],\n",
              "    num_rows: 41584\n",
              "})"
            ]
          },
          "execution_count": 2,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "from datasets import load_dataset\n",
        "\n",
        "data = load_dataset(\"jamescalam/ai-arxiv-chunked\", split=\"train\")\n",
        "data"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "wrC-XHrTDhG6"
      },
      "source": [
        "We have 41.5K chunks, where each chunk is roughly the length of 1-2 paragraphs in length. Here is an example of a single record:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 3,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "RQg8wiUQ4bOk",
        "outputId": "90a416c3-a60d-474e-e643-77a64ff91913"
      },
      "outputs": [
        {
          "data": {
            "text/plain": [
              "{'doi': '1910.01108',\n",
              " 'chunk-id': '0',\n",
              " 'chunk': 'DistilBERT, a distilled version of BERT: smaller,\\nfaster, cheaper and lighter\\nVictor SANH, Lysandre DEBUT, Julien CHAUMOND, Thomas WOLF\\nHugging Face\\n{victor,lysandre,julien,thomas}@huggingface.co\\nAbstract\\nAs Transfer Learning from large-scale pre-trained models becomes more prevalent\\nin Natural Language Processing (NLP), operating these large models in on-theedge and/or under constrained computational training or inference budgets remains\\nchallenging. In this work, we propose a method to pre-train a smaller generalpurpose language representation model, called DistilBERT, which can then be \ufb01netuned with good performances on a wide range of tasks like its larger counterparts.\\nWhile most prior work investigated the use of distillation for building task-speci\ufb01c\\nmodels, we leverage knowledge distillation during the pre-training phase and show\\nthat it is possible to reduce the size of a BERT model by 40%, while retaining 97%\\nof its language understanding capabilities and being 60% faster. To leverage the\\ninductive biases learned by larger models during pre-training, we introduce a triple\\nloss combining language modeling, distillation and cosine-distance losses. Our\\nsmaller, faster and lighter model is cheaper to pre-train and we demonstrate its',\n",
              " 'id': '1910.01108',\n",
              " 'title': 'DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter',\n",
              " 'summary': 'As Transfer Learning from large-scale pre-trained models becomes more\\nprevalent in Natural Language Processing (NLP), operating these large models in\\non-the-edge and/or under constrained computational training or inference\\nbudgets remains challenging. In this work, we propose a method to pre-train a\\nsmaller general-purpose language representation model, called DistilBERT, which\\ncan then be fine-tuned with good performances on a wide range of tasks like its\\nlarger counterparts. While most prior work investigated the use of distillation\\nfor building task-specific models, we leverage knowledge distillation during\\nthe pre-training phase and show that it is possible to reduce the size of a\\nBERT model by 40%, while retaining 97% of its language understanding\\ncapabilities and being 60% faster. To leverage the inductive biases learned by\\nlarger models during pre-training, we introduce a triple loss combining\\nlanguage modeling, distillation and cosine-distance losses. Our smaller, faster\\nand lighter model is cheaper to pre-train and we demonstrate its capabilities\\nfor on-device computations in a proof-of-concept experiment and a comparative\\non-device study.',\n",
              " 'source': 'http://arxiv.org/pdf/1910.01108',\n",
              " 'authors': ['Victor Sanh',\n",
              "  'Lysandre Debut',\n",
              "  'Julien Chaumond',\n",
              "  'Thomas Wolf'],\n",
              " 'categories': ['cs.CL'],\n",
              " 'comment': 'February 2020 - Revision: fix bug in evaluation metrics, updated\\n  metrics, argumentation unchanged. 5 pages, 1 figure, 4 tables. Accepted at\\n  the 5th Workshop on Energy Efficient Machine Learning and Cognitive Computing\\n  - NeurIPS 2019',\n",
              " 'journal_ref': None,\n",
              " 'primary_category': 'cs.CL',\n",
              " 'published': '20191002',\n",
              " 'updated': '20200301',\n",
              " 'references': [{'id': '1910.01108'}]}"
            ]
          },
          "execution_count": 3,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "data[0]"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "euFtJiIz4bOk"
      },
      "source": [
        "Format the data into the format we need, this will contain `id`, `text` (which we will embed), and `metadata`. For this use-case we don't need metadata but it can be useful to include so that if needed in the future we can make use of metadata filtering."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 4,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "u-svyAMw4bOl",
        "outputId": "61f6e9af-2998-4e52-a216-a912473106b7"
      },
      "outputs": [
        {
          "data": {
            "text/plain": [
              "Dataset({\n",
              "    features: ['id', 'text', 'metadata'],\n",
              "    num_rows: 41584\n",
              "})"
            ]
          },
          "execution_count": 4,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "data = data.map(lambda x: {\n",
        "    \"id\": f'{x[\"id\"]}-{x[\"chunk-id\"]}',\n",
        "    \"text\": x[\"chunk\"],\n",
        "    \"metadata\": {\n",
        "        \"title\": x[\"title\"],\n",
        "        \"url\": x[\"source\"],\n",
        "        \"primary_category\": x[\"primary_category\"],\n",
        "        \"published\": x[\"published\"],\n",
        "        \"updated\": x[\"updated\"],\n",
        "        \"text\": x[\"chunk\"],\n",
        "    }\n",
        "})\n",
        "# drop uneeded columns\n",
        "data = data.remove_columns([\n",
        "    \"title\", \"summary\", \"source\",\n",
        "    \"authors\", \"categories\", \"comment\",\n",
        "    \"journal_ref\", \"primary_category\",\n",
        "    \"published\", \"updated\", \"references\",\n",
        "    \"doi\", \"chunk-id\",\n",
        "    \"chunk\"\n",
        "])\n",
        "data"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "MYzwm_q_4bOl"
      },
      "source": [
        "We need to define an embedding model to create our embedding vectors for retrieval, for that we will be using OpenAI's text-embedding-ada-002. There is some cost associated with this model, so be aware of that (costs for running this notebook are <$1)."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 5,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "nVjJ6gGd4bOl",
        "outputId": "b4e4ad45-ff8b-4cc6-b667-9553ebf150d9"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Enter your OpenAI API key: \u00b7\u00b7\u00b7\u00b7\u00b7\u00b7\u00b7\u00b7\u00b7\u00b7\n"
          ]
        }
      ],
      "source": [
        "import os\n",
        "import openai\n",
        "import getpass  # platform.openai.com\n",
        "\n",
        "# get API key from top-right dropdown on OpenAI website\n",
        "openai.api_key = os.getenv(\"OPENAI_API_KEY\") or getpass.getpass(\"Enter your OpenAI API key: \")\n",
        "\n",
        "embed_model = \"text-embedding-ada-002\""
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ndAuMyYC4bOm"
      },
      "source": [
        "Now we create our vector DB to store our vectors. For this we need to get a [free Pinecone API key](https://app.pinecone.io) \u2014 the API key can be found in the \"API Keys\" button found in the left navbar of the Pinecone dashboard."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 6,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "ThmEhCcI4bOm",
        "outputId": "6a0bf8a7-4751-456f-87fe-a0cce74228d7"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "\u00b7\u00b7\u00b7\u00b7\u00b7\u00b7\u00b7\u00b7\u00b7\u00b7\n"
          ]
        }
      ],
      "source": [
        "from pinecone import Pinecone\n",
        "\n",
        "# initialize connection to pinecone (get API key at app.pinecone.io)\n",
        "api_key = os.getenv(\"PINECONE_API_KEY\") or getpass.getpass()\n",
        "\n",
        "# configure client\n",
        "pc = Pinecone(api_key=api_key)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "YtEcZE5AfQHW"
      },
      "source": [
        "Now we setup our index specification, this allows us to define the cloud provider and region where we want to deploy our index. You can find a list of all [available providers and regions here](https://docs.pinecone.io/docs/projects)."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 7,
      "metadata": {
        "id": "vVmlAytrfeUJ"
      },
      "outputs": [],
      "source": [
        "from pinecone import ServerlessSpec\n",
        "\n",
        "spec = ServerlessSpec(\n",
        "    cloud=\"aws\", region=\"us-west-2\"\n",
        ")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "-nu2KHWG4bOm"
      },
      "source": [
        "Creating an index, we set `dimension` equal to to dimensionality of Ada-002 (`1536`), and use a `metric` also compatible with Ada-002 (this can be either `cosine` or `dotproduct`). We also pass our `spec` to index initialization."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 9,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "D4E9wrzx4bOm",
        "outputId": "672505f4-fd29-4440-b4a3-d3da248e6866"
      },
      "outputs": [
        {
          "data": {
            "text/plain": [
              "{'dimension': 1536,\n",
              " 'index_fullness': 0.0,\n",
              " 'namespaces': {},\n",
              " 'total_vector_count': 0}"
            ]
          },
          "execution_count": 9,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "import time\n",
        "\n",
        "index_name = \"rerankers\"\n",
        "existing_indexes = [\n",
        "    index_info[\"name\"] for index_info in pc.list_indexes()\n",
        "]\n",
        "\n",
        "# check if index already exists (it shouldn't if this is first time)\n",
        "if index_name not in existing_indexes:\n",
        "    # if does not exist, create index\n",
        "    pc.create_index(\n",
        "        index_name,\n",
        "        dimension=1536,  # dimensionality of ada 002\n",
        "        metric='dotproduct',\n",
        "        spec=spec\n",
        "    )\n",
        "    # wait for index to be initialized\n",
        "    while not pc.describe_index(index_name).status['ready']:\n",
        "        time.sleep(1)\n",
        "\n",
        "# connect to index\n",
        "index = pc.Index(index_name)\n",
        "time.sleep(1)\n",
        "# view index stats\n",
        "index.describe_index_stats()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "H_ujDG1G8nxu"
      },
      "source": [
        "Define embedding function with OpenAI:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 10,
      "metadata": {
        "id": "ZNw4zxav8sGT"
      },
      "outputs": [],
      "source": [
        "def embed(batch: list[str]) -> list[float]:\n",
        "    # create embeddings (exponential backoff to avoid RateLimitError)\n",
        "    for j in range(5):  # max 5 retries\n",
        "        try:\n",
        "            res = openai.embeddings.create(\n",
        "                input=batch[\"text\"],\n",
        "                model=embed_model\n",
        "            )\n",
        "            passed = True\n",
        "        except openai.RateLimitError:\n",
        "            time.sleep(2**j)  # wait 2^j seconds before retrying\n",
        "            print(\"Retrying...\")\n",
        "    if not passed:\n",
        "        raise RuntimeError(\"Failed to create embeddings.\")\n",
        "    # get embeddings\n",
        "    embeds = [record.embedding for record in res.data]\n",
        "    return embeds"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ZI76rcTi4bOm"
      },
      "source": [
        "We can see the index is currently empty with a `total_vector_count` of `0`. We can begin populating it with OpenAI's `text-embedding-ada-002` built embeddings like so:\n",
        "\n",
        "**\u26a0\ufe0f WARNING: Embedding costs for the full dataset as of 3 Jan 2024 is ~$5.70**"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 11,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 49,
          "referenced_widgets": [
            "8a157731fb84424785cba10bd64d450c",
            "b34426ef40e24f7f8592f9456472c603",
            "9200a9b812ef4aed8b1cfd785f96453d",
            "d4de2a2b84ec458aa7fb1a8b81c480f4",
            "86e2de3b65d64976a855d756790692d3",
            "547845fccd68443f88d66ba4eaf46d76",
            "0cbd4c3eb5f94ca78f9956c21d4e44f2",
            "7da2ca553f8f4e5e8d23d5e86f9cda1b",
            "f9d766d45c42447eb5625655ccf313fc",
            "85157399efd74dfdb679f4e9912b782e",
            "00b73e22d7a042e6aea29cf1c1719ebf"
          ]
        },
        "id": "a2xvoFt04bOn",
        "outputId": "36e3b293-f9b7-439a-f72b-fa55b8b18f84"
      },
      "outputs": [
        {
          "data": {
            "application/vnd.jupyter.widget-view+json": {
              "model_id": "8a157731fb84424785cba10bd64d450c",
              "version_major": 2,
              "version_minor": 0
            },
            "text/plain": [
              "  0%|          | 0/416 [00:00<?, ?it/s]"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        }
      ],
      "source": [
        "from tqdm.auto import tqdm\n",
        "\n",
        "batch_size = 100  # how many embeddings we create and insert at once\n",
        "\n",
        "for i in tqdm(range(0, len(data), batch_size)):\n",
        "    passed = False\n",
        "    # find end of batch\n",
        "    i_end = min(len(data), i+batch_size)\n",
        "    # create batch\n",
        "    batch = data[i:i_end]\n",
        "    embeds = embed(batch[\"text\"])\n",
        "    to_upsert = list(zip(batch[\"id\"], embeds, batch[\"metadata\"]))\n",
        "    # upsert to Pinecone\n",
        "    index.upsert(vectors=to_upsert)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "oyFZKhUa4bOn"
      },
      "source": [
        "Now let's test retrieval _without_ Cohere's reranking model."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 12,
      "metadata": {
        "id": "6pUo5EQK4bOn"
      },
      "outputs": [],
      "source": [
        "def get_docs(query: str, top_k: int) -> list[str]:\n",
        "    # encode query\n",
        "    xq = embed([query])[0]\n",
        "    # search pinecone index\n",
        "    res = index.query(vector=xq, top_k=top_k, include_metadata=True)\n",
        "    # get doc text\n",
        "    docs = {x[\"metadata\"]['text']: i for i, x in enumerate(res[\"matches\"])}\n",
        "    return docs"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 13,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "C3FASr-04bOn",
        "outputId": "b18090e4-35d4-4ba2-d8fe-afb2f2968c2b"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Bill Yuchen Lin, Wangchunshu Zhou, Ming Shen, Pei Zhou, Chandra Bhagavatula, Yejin Choi,\n",
            "and Xiang Ren. CommonGen: A constrained text generation challenge for generative commonsense reasoning. In Findings of the Association for Computational Linguistics: EMNLP\n",
            "2020 , pp. 1823\u20131840, Online, November 2020. Association for Computational Linguistics.\n",
            "doi: 10.18653/v1/2020.\ufb01ndings-emnlp.165. URL https://aclanthology.org/2020.\n",
            "findings-emnlp.165 .\n",
            "Alisa Liu, Maarten Sap, Ximing Lu, Swabha Swayamdipta, Chandra Bhagavatula, Noah A. Smith,\n",
            "and Yejin Choi. DExperts: Decoding-time controlled text generation with experts and antiexperts. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume\n",
            "1: Long Papers) , pp. 6691\u20136706, Online, August 2021. Association for Computational Linguistics. doi: 10.18653/v1/2021.acl-long.522. URL https://aclanthology.org/2021.\n",
            "acl-long.522 .\n",
            "---\n",
            "2017. Asian Federation of Natural Language Processing. URL https://aclanthology.\n",
            "org/I17-1099 .\n",
            "Bill Yuchen Lin, Wangchunshu Zhou, Ming Shen, Pei Zhou, Chandra Bhagavatula, Yejin Choi, and\n",
            "Xiang Ren. CommonGen: A constrained text generation challenge for generative commonsense\n",
            "reasoning. In Findings of the Association for Computational Linguistics: EMNLP 2020 , pp.\n",
            "1823\u20131840, Online, 2020. Association for Computational Linguistics. doi: 10.18653/v1/2020.\n",
            "\ufb01ndings-emnlp.165.\n",
            "Chin-Yew Lin. Rouge: A package for automatic evaluation of summaries. In Text summarization\n",
            "branches out , pp. 74\u201381, 2004.\n",
            "Siqi Liu, Zhenhai Zhu, Ning Ye, Sergio Guadarrama, and Kevin Murphy. Improved image captioning\n",
            "via policy gradient optimization of spider. In Proceedings of the IEEE international conference on\n",
            "computer vision , pp. 873\u2013881, 2017.\n",
            "Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike\n",
            "---\n",
            "forComputational Linguistics: ACL-IJCNLP 2021,\n",
            "pages 596\u2013610, Online. Association for Computational Linguistics.\n",
            "Bill Yuchen Lin, Wangchunshu Zhou, Ming Shen,\n",
            "Pei Zhou, Chandra Bhagavatula, Yejin Choi, and\n",
            "Xiang Ren. 2020. CommonGen: A constrained\n",
            "text generation challenge for generative commonsense reasoning. In Findings oftheAssociation\n",
            "forComputational Linguistics: EMNLP 2020, pages\n",
            "1823\u20131840, Online. Association for Computational\n",
            "Linguistics.\n",
            "Chin-Yew Lin. 2004. Rouge: A package for automatic\n",
            "evaluation of summaries. In Text summarization\n",
            "branches out, pages 74\u201381.\n",
            "Wang Ling, Dani Yogatama, Chris Dyer, and Phil Blunsom. 2017. Program induction by rationale generation: Learning to solve and explain algebraic\n",
            "word problems. In Proceedings ofthe55th Annual\n",
            "Meeting ofthe Association for Computational\n",
            "Linguistics (V olume 1:Long Papers), pages 158\u2013\n",
            "167, Vancouver, Canada. Association for Computational Linguistics.\n",
            "Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis,\n",
            "---\n",
            "Haoran Li, Abhinav Arora, Shuohui Chen, Anchit Gupta, Sonal Gupta, and Yashar Mehdad. Mtop: A\n",
            "comprehensivemultilingual task-oriented semanticparsing benchmark. arXiv preprintarXiv:2008.09335 ,\n",
            "2020.\n",
            "BillYuchenLin,WangchunshuZhou,MingShen,PeiZhou,ChandraBhagavatula,YejinChoi,andXiangRen.\n",
            "CommonGen: A constrained text generation challenge for generative commonsense reasoning. In Findings\n",
            "oftheAssociationforComputationalLinguistics: EMNLP2020 ,pp.1823\u20131840,Online,November2020.Association for Computational Linguistics. URL https://www.aclweb.org/anthology/2020.findings-emnlp.\n",
            "165.\n",
            "Wang Ling, Dani Yogatama, Chris Dyer, and Phil Blunsom. Program induction by rationale generation:\n",
            "Learning to solve and explain algebraic word problems. ACL, 2017. doi: 10.18653/v1/P17-1015. URL\n",
            "https://aclanthology.org/P17-1015 .\n",
            "30\n",
            "---\n",
            "Hung Le, Yue Wang, Akhilesh Deepak Gotmare, Silvio Savarese, and Steven C. H. Hoi. 2022b.\n",
            "Coderl: Mastering code generation through pretrained models and deep reinforcement learning.\n",
            "ArXiv , abs/2207.01780.\n",
            "Juncen Li, Robin Jia, He He, and Percy Liang. 2018. Delete, retrieve, generate: a simple approach to\n",
            "sentiment and style transfer. In Proceedings of the 2018 Conference of the North American Chapter\n",
            "of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long\n",
            "Papers) , pages 1865\u20131874, New Orleans, Louisiana. Association for Computational Linguistics.\n",
            "Bill Yuchen Lin, Wangchunshu Zhou, Ming Shen, Pei Zhou, Chandra Bhagavatula, Yejin Choi,\n",
            "and Xiang Ren. 2020. CommonGen: A constrained text generation challenge for generative\n",
            "commonsense reasoning. In Findings of the Association for Computational Linguistics: EMNLP\n",
            "2020 , pages 1823\u20131840, Online. Association for Computational Linguistics.\n",
            "Jiacheng Liu, Skyler Hallinan, Ximing Lu, Pengfei He, Sean Welleck, Hannaneh Hajishirzi, and Yejin\n",
            "---\n",
            "pre-trained language models for dialogue adaptation, 2020.\n",
            "Ya Li, Xinmei Tian, Tongliang Liu, and Dacheng Tao. On better exploring and exploiting task relationships in multitask learning: Joint model and feature learning. IEEE Transactions on Neural\n",
            "Networks and Learning Systems , 29(5):1975\u20131985, May 2018. ISSN 2162-2388. doi: 10.1109/\n",
            "tnnls.2017.2690683. URL http://dx.doi.org/10.1109/TNNLS.2017.2690683 .\n",
            "Bill Yuchen Lin, Wangchunshu Zhou, Ming Shen, Pei Zhou, Chandra Bhagavatula, Yejin Choi, and\n",
            "Xiang Ren. CommonGen: A constrained text generation challenge for generative commonsense\n",
            "reasoning. In Findings of the Association for Computational Linguistics: EMNLP 2020 , pp.\n",
            "1823\u20131840, Online, November 2020. Association for Computational Linguistics. URL https:\n",
            "//www.aclweb.org/anthology/2020.findings-emnlp.165 .\n",
            "Yu-Hsiang Lin, Chian-Yu Chen, Jean Lee, Zirui Li, Yuyan Zhang, Mengzhou Xia, Shruti Rijhwani,\n",
            "---\n",
            "from large language models make small reasoners\n",
            "better. ArXiv preprint , abs/2210.06726.\n",
            "Yifei Li, Zeqi Lin, Shizhuo Zhang, Qiang Fu, Bei Chen,\n",
            "Jian-Guang Lou, and Weizhu Chen. 2022b. On the\n",
            "advance of making language models better reasoners.\n",
            "ArXiv preprint , abs/2206.02336.\n",
            "Bill Yuchen Lin, Wangchunshu Zhou, Ming Shen, Pei\n",
            "Zhou, Chandra Bhagavatula, Yejin Choi, and Xiang\n",
            "Ren. 2020. CommonGen: A constrained text generation challenge for generative commonsense reasoning. In Findings of the Association for Computational Linguistics: EMNLP 2020 , pages 1823\u20131840,\n",
            "Online. Association for Computational Linguistics.\n",
            "Wang Ling, Dani Yogatama, Chris Dyer, and Phil Blunsom. 2017. Program induction by rationale generation: Learning to solve and explain algebraic word\n",
            "problems. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics\n",
            "(Volume 1: Long Papers) , pages 158\u2013167, Vancouver,\n",
            "Canada. Association for Computational Linguistics.\n",
            "Chenzhengyi Liu, Jie Huang, Kerui Zhu, and Kevin\n",
            "---\n",
            "Canada. Association for Computational Linguistics.\n",
            "Chenzhengyi Liu, Jie Huang, Kerui Zhu, and Kevin\n",
            "Chen-Chuan Chang. 2022a. Dimongen: Diversified generative commonsense reasoning for explaining concept relationships. ArXiv preprint ,\n",
            "abs/2212.10545.\n",
            "Jiachang Liu, Dinghan Shen, Yizhe Zhang, Bill Dolan,\n",
            "Lawrence Carin, and Weizhu Chen. 2022b. What\n",
            "makes good in-context examples for GPT-3? In\n",
            "Proceedings of Deep Learning Inside Out (DeeLIO\n",
            "2022): The 3rd Workshop on Knowledge Extraction and Integration for Deep Learning Architectures ,\n",
            "pages 100\u2013114, Dublin, Ireland and Online. Association for Computational Linguistics.\n",
            "Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis,\n",
            "Luke Zettlemoyer, and Veselin Stoyanov. 2019.\n",
            "Roberta: A robustly optimized bert pretraining approach. ArXiv preprint , abs/1907.11692.\n",
            "Pan Lu, Swaroop Mishra, Tony Xia, Liang Qiu, KaiWei Chang, Song-Chun Zhu, Oyvind Tafjord, Peter\n",
            "---\n",
            "Alisa Liu, Maarten Sap, Ximing Lu, Swabha\n",
            "Swayamdipta, Chandra Bhagavatula, Noah A.\n",
            "Smith, and Yejin Choi. 2021a. DExperts: Decodingtime controlled text generation with experts and antiexperts. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguisticsand the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers) ,\n",
            "pages 6691\u20136706, Online. Association for Computational Linguistics.\n",
            "Haochen Liu, Jamell Dacon, Wenqi Fan, Hui Liu, Zitao\n",
            "Liu, and Jiliang Tang. 2020. Does gender matter?\n",
            "towards fairness in dialogue systems. In Proceedings of the 28th International Conference on Computational Linguistics , pages 4403\u20134416, Barcelona,\n",
            "Spain (Online). International Committee on Computational Linguistics.\n",
            "Pengfei Liu, Weizhe Yuan, Jinlan Fu, Zhengbao Jiang,\n",
            "Hiroaki Hayashi, and Graham Neubig. 2023. Pretrain, prompt, and predict: A systematic survey of\n",
            "prompting methods in natural language processing.\n",
            "ACM Comput. Surv. , 55(9).\n",
            "Qi Liu, Dani Yogatama, and Phil Blunsom. 2022.\n",
            "---\n",
            "MRC framework for named entity recognition. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics , pp. 5849\u20135859, 2020. URL https:\n",
            "//aclanthology.org/2020.acl-main.519 .\n",
            "16\n",
            "Published as a conference paper at ICLR 2022\n",
            "Xin Li and Dan Roth. Learning question classi\ufb01ers. In COLING 2002: The 19th International Conference on Computational Linguistics , 2002. URL https://www.aclweb.org/anthology/\n",
            "C02-1150 .\n",
            "Bill Yuchen Lin, Wangchunshu Zhou, Ming Shen, Pei Zhou, Chandra Bhagavatula, Yejin Choi, and\n",
            "Xiang Ren. CommonGen: A constrained text generation challenge for generative commonsense\n",
            "reasoning. In Findings of the Association for Computational Linguistics: EMNLP 2020 , pp.\n",
            "1823\u20131840, 2020. URL https://aclanthology.org/2020.findings-emnlp.165 .\n",
            "Han Liu, Xiaotong Zhang, Lu Fan, Xuandi Fu, Qimai Li, Xiao-Ming Wu, and Albert Y .S. Lam.\n",
            "Reconstructing capsule networks for zero-shot intent classi\ufb01cation. In Proceedings of the 2019\n",
            "---\n",
            "commongen sentence generation (Lin et al., 2020)\n",
            "sciq answer generation (Welbl et al., 2017)\n",
            "openbookqa question answering (Mihaylov et al., 2018)\n",
            "Texual\n",
            "Entailment1386\n",
            "1387\n",
            "1388\n",
            "1344anli r2 entailment (Williams et al., 2022)\n",
            "anli r3 entailment (Williams et al., 2022)\n",
            "cb entailment (Wang et al., 2019)\n",
            "glue entailment classi\ufb01cation (Wang et al., 2018)\n",
            "Mathematics104\n",
            "119\n",
            "697semeval closed vocabulary math answer generation (Hopkins et al., 2019)\n",
            "semeval geometric math answer generation (Hopkins et al., 2019)\n",
            "mmmlu answer generation formal logic (Hendrycks et al., 2021a)\n",
            "Abductive\n",
            "Reasoning332 tellmewhy answer generation (Lal et al., 2021)\n",
            "Spatial\n",
            "Reasoning83\n",
            "80\n",
            "151babi t1 single supporting fact answer generation (Weston et al., 2015)\n",
            "piqa answer generation (Bisk et al., 2020)\n",
            "tomqa \ufb01nd location easy clean (Nematzadeh et al., 2018)\n",
            "Analogical\n",
            "Reasoning102\n",
            "1152commongen sentence generation (Lin et al., 2020)\n",
            "bard analogical reasoning causation (Fulda et al., 2017)\n",
            "---\n",
            "movie, and this film is one of those films where he can be convincing in it (and his\n",
            "trademark acting, as you can see in the\n",
            "27\n",
            "Published as a conference paper at ICLR 2023\n",
            "B.4 C OMMON GEN\n",
            "B.4.1 S ETUP\n",
            "CommonGen (Lin et al., 2020) deals with task of generating coherent sentences describing an\n",
            "input set of concepts (eg. \"a man is throwing a frisbee\"). For training RL methods, we consider\n",
            "3 traditional lexical rewards namely Rouge-1, Rouge-avg (which is an average of Rouge-1, 2 and\n",
            "L) and meteor. Additionally, we also train with task-speci\ufb01c rewards such as CIDEr (Vedantam\n",
            "et al., 2015), SPICE (Anderson et al., 2016) and SPiDer (Liu et al., 2017) which is a just a linear\n",
            "combination of both with equal weights. We chose T5-base as the base LM since it is well-suited\n",
            "for structure to text tasks. We additionally note that concept set inputs are pre\ufb01xed with \"generate a\n",
            "sentence with:\" to encourage exploration.\n",
            "During our initial experiments when \ufb01ne-tuning directly on LM, we observed that policy learns to\n",
            "repeat the prompted concepts in order to maximize rewards resulting in a well-known problem of\n",
            "---\n",
            "Sharon Levy, Michael Saxon, and William Yang Wang.\n",
            "2021. The truth is out there: Investigating conspiracy theories in text generation. In Findings of The\n",
            "Joint Conference of the 59th Annual Meeting of the\n",
            "Association for Computational Linguistics and the\n",
            "11th International Joint Conference on Natural Language Processing .\n",
            "Alisa Liu, Maarten Sap, Ximing Lu, Swabha\n",
            "Swayamdipta, Chandra Bhagavatula, Noah A.\n",
            "Smith, and Yejin Choi. 2021. DExperts: Decodingtime controlled text generation with experts and antiexperts. In Proceedings of the 59th Annual Meeting\n",
            "of the Association for Computational Linguistics .\n",
            "Haochen Liu, Jamell Dacon, Wenqi Fan, Hui Liu, Zitao\n",
            "Liu, and Jiliang Tang. 2020a. Does gender matter?\n",
            "towards fairness in dialogue systems. In Proceedings of the 28th International Conference on Computational Linguistics , pages 4403\u20134416, Barcelona,\n",
            "Spain (Online). International Committee on Computational Linguistics.\n",
            "Haochen Liu, Wentao Wang, Yiqi Wang, Hui Liu, Zitao Liu, and Jiliang Tang. 2020b. Mitigating gender\n",
            "bias for neural dialogue generation with adversarial\n",
            "learning. In Proceedings of the 2020 Conference on\n",
            "---\n",
            "Bill Yuchen Lin, Ziyi Wu, Yichi Yang, Dong-Ho Lee,\n",
            "and Xiang Ren. 2021. RiddleSense: Reasoning\n",
            "about riddle questions featuring linguistic creativity and commonsense knowledge. In Findings of\n",
            "the Association for Computational Linguistics: ACLIJCNLP 2021 , pages 1504\u20131515, Online. Association for Computational Linguistics.\n",
            "Jiacheng Liu, Alisa Liu, Ximing Lu, Sean Welleck, Peter West, Ronan Le Bras, Yejin Choi, and Hannaneh\n",
            "Hajishirzi. 2022. Generated knowledge prompting\n",
            "for commonsense reasoning. In Proceedings of the\n",
            "60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , pages\n",
            "3154\u20133169, Dublin, Ireland. Association for Computational Linguistics.\n",
            "Nicholas Lourie, Ronan Le Bras, Chandra Bhagavatula,\n",
            "and Yejin Choi. 2021. Unicorn on rainbow: A universal commonsense reasoning model on a new multitask benchmark. In Proceedings of the AAAI Conference on Arti\ufb01cial Intelligence , volume 35, pages\n",
            "13480\u201313488.\n",
            "Ximing Lu, Sean Welleck, Liwei Jiang, Jack Hessel,\n",
            "Lianhui Qin, Peter West, Prithviraj Ammanabrolu,\n",
            "---\n",
            "of the Association for Computational Linguistics (Volume 1: Long Papers) . Association for Computational Linguistics, 2022.\n",
            "Nasrin Mostafazadeh, Nathanael Chambers, Xiaodong He, Devi Parikh, Dhruv Batra, Lucy Vanderwende, Pushmeet Kohli, and James Allen. A corpus and cloze evaluation for deeper understanding\n",
            "of commonsense stories. In Proceedings of the 2016 Conference of the North American Chapter\n",
            "of the Association for Computational Linguistics: Human Language Technologies . Association\n",
            "for Computational Linguistics, 2016.\n",
            "Yixin Nie, Adina Williams, Emily Dinan, Mohit Bansal, Jason Weston, and Douwe Kiela. Adversarial NLI: A new benchmark for natural language understanding. In Proceedings of the 58th\n",
            "Annual Meeting of the Association for Computational Linguistics . Association for Computational\n",
            "Linguistics, 2020.\n",
            "Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L Wainwright, Pamela Mishkin, Chong\n",
            "Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow\n",
            "instructions with human feedback. arXiv preprint arXiv:2203.02155 , 2022.\n",
            "---\n",
            "Jonathan Berant. 2019. CommonsenseQA: A question answering challenge targeting commonsense\n",
            "knowledge. In Proceedings of the 2019 Conference\n",
            "of the North American Chapter of the Association\n",
            "for Computational Linguistics: Human Language\n",
            "Technologies, Volume 1 (Long and Short Papers) ,\n",
            "pages 4149\u20134158, Minneapolis, Minnesota. Association for Computational Linguistics.\n",
            "Wenya Wang, Vivek Srikumar, Hanna Hajishirzi, and\n",
            "Noah A Smith. 2022. Elaboration-generating commonsense question answering at scale. arXiv\n",
            "preprint arXiv:2209.01232 .\n",
            "Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten\n",
            "Bosma, Ed Chi, Quoc Le, and Denny Zhou. 2022.\n",
            "Chain of thought prompting elicits reasoning in large\n",
            "language models. arXiv preprint arXiv:2201.11903 .\n",
            "Yonghui Wu, Mike Schuster, Zhifeng Chen, Quoc V\n",
            "Le, Mohammad Norouzi, Wolfgang Macherey,\n",
            "Maxim Krikun, Yuan Cao, Qin Gao, Klaus\n",
            "Macherey, et al. 2016. Google\u2019s neural machine\n",
            "translation system: Bridging the gap between human and machine translation. arXiv preprint\n",
            "arXiv:1609.08144 .\n",
            "---\n",
            "Veselin Stoyanov, and Luke Zettlemoyer. BART: denoising sequence-to-sequence pre-training for\n",
            "natural language generation, translation, and comprehension. In ACL, pages 7871\u20137880, 2020.\n",
            "Xin Li and Dan Roth. Learning question classi\ufb01ers. In COLING , 2002.\n",
            "Xiujun Li, Xi Yin, Chunyuan Li, Pengchuan Zhang, Xiaowei Hu, Lei Zhang, Lijuan Wang, Houdong\n",
            "Hu, Li Dong, Furu Wei, et al. Oscar: Object-semantics aligned pre-training for vision-language\n",
            "tasks. In ECCV , pages 121\u2013137, 2020.\n",
            "Bill Yuchen Lin, Wangchunshu Zhou, Ming Shen, Pei Zhou, Chandra Bhagavatula, Yejin Choi, and\n",
            "Xiang Ren. CommonGen: A constrained text generation challenge for generative commonsense\n",
            "reasoning. In Findings of the Association for Computational Linguistics: EMNLP 2020 , November\n",
            "2020.\n",
            "Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr\n",
            "Doll\u00e1r, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In ECCV , pages\n",
            "740\u2013755, 2014.\n",
            "---\n",
            "the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on\n",
            "Natural Language Processing (Volume 1: Long Papers) . Association for Computational Linguistics, Online, 4582\u20134597.\n",
            "https://doi.org/10.18653/v1/2021.acl-long.353\n",
            "[72] Xiang Lisa Li, John Thickstun, Ishaan Gulrajani, Percy Liang, and Tatsunori B. Hashimoto. 2022. Diffusion-LM\n",
            "Improves Controllable Text Generation. https://doi.org/10.48550/ARXIV.2205.14217\n",
            "[73] Alisa Liu, Maarten Sap, Ximing Lu, Swabha Swayamdipta, Chandra Bhagavatula, Noah A. Smith, and Yejin Choi.\n",
            "2021. DExperts: Decoding-Time Controlled Text Generation with Experts and Anti-Experts. In Proceedings of the\n",
            "59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on\n",
            "Natural Language Processing (Volume 1: Long Papers) . Association for Computational Linguistics, Online, 6691\u20136706.\n",
            "https://doi.org/10.18653/v1/2021.acl-long.522\n",
            "---\n",
            "I. Sutskever. 2018. Improving language understanding by generative pre-training. Technical Report,\n",
            "OpenAI .\n",
            "P. Rajpurkar, J. Zhang, K. Lopyrev, and P. Liang. 2016.\n",
            "Squad: 100,000+ questions for machine comprehension of text. In Empirical Methods in Natural Language Processing (EMNLP) .\n",
            "M. Roemmele, C. Bejan, and A. Gordon. 2011. Choice\n",
            "of plausible alternatives: An evaluation of commonsense causal reasoning. In AAAI Spring Symposium\n",
            "on Logical Formalizations of Commonsense Reasoning .Roy Schwartz, Maarten Sap, Ioannis Konstas, Leila\n",
            "Zilles, Yejin Choi, and Noah A. Smith. 2017. The\n",
            "effect of different writing tasks on linguistic style:\n",
            "A case study of the roc story cloze task. In CoNLL .\n",
            "M. Seo, A. Kembhavi, A. Farhadi, and H. Hajishirzi.\n",
            "2016. Bidirectional attention \ufb02ow for machine comprehension. arXiv .\n",
            "Robert Speer, Joshua Chin, and Catherine Havasi.\n",
            "2017. Conceptnet 5.5: An open multilingual graph\n",
            "of general knowledge. In AAAI , pages 4444\u20134451.\n",
            "---\n",
            "Meeting of the Association for Computational Linguistics , pages 5185\u20135198, Online. Association for\n",
            "Computational Linguistics.\n",
            "Prajjwal Bhargava and Vincent Ng. 2022. Commonsense knowledge reasoning and generation with pretrained language models: A survey. Proceedings of\n",
            "the AAAI Conference on Artificial Intelligence .\n",
            "Rishi Bommasani, Drew A Hudson, Ehsan Adeli,\n",
            "Russ Altman, Simran Arora, Sydney von Arx,\n",
            "Michael S Bernstein, Jeannette Bohg, Antoine Bosselut, Emma Brunskill, et al. 2021. On the opportunities and risks of foundation models. ArXiv preprint ,\n",
            "abs/2108.07258.\n",
            "Hugo Bronkhorst, Gerrit Roorda, Cor Suhre, and Martin Goedhart. 2020. Logical reasoning in formal\n",
            "and everyday reasoning tasks. International Journal\n",
            "of Science and Mathematics Education , 18(8):1673\u2013\n",
            "1694.\n",
            "Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie\n",
            "Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind\n",
            "Neelakantan, Pranav Shyam, Girish Sastry, Amanda\n",
            "Askell, Sandhini Agarwal, Ariel Herbert-V oss,\n",
            "Gretchen Krueger, Tom Henighan, Rewon Child,\n",
            "---\n",
            "significant for 3 out of 7 datasets.\n",
            "28\n",
            "K New Tasks\n",
            "Constrained Generation We introduce \u201cCommonGen-Hard,\" a more challenging extension of the\n",
            "CommonGen dataset (Lin et al., 2020), designed to test state-of-the-art language models\u2019 advanced\n",
            "commonsense reasoning, contextual understanding, and creative problem-solving. CommonGenHard requires models to generate coherent sentences incorporating 20-30 concepts, rather than only\n",
            "the 3-5 related concepts given in CommonGen. SELF-REFINE focuses on iterative creation with\n",
            "introspective feedback, making it suitable for evaluating the effectiveness of language models on the\n",
            "CommonGen-Hard task.\n",
            "Acronym Generation Acronym generation requires an iterative refinement process to create\n",
            "concise and memorable representations of complex terms or phrases, involving tradeoffs between\n",
            "length, ease of pronunciation, and relevance, and thus serves as a natural testbed for our approach.\n",
            "We source a dataset of 250 acronyms4and manually prune it to remove offensive or uninformative\n",
            "acronyms.\n",
            "L Code Readability\n",
            "Orthogonal to the correctness, readability is another important quality of a piece of code: though not\n",
            "related to the execution results of the code, code readability may significantly affect the usability,\n",
            "upgradability, and ease of maintenance of an entire codebase. In this section, we consider the problem\n",
            "---\n",
            "AAAI Spring Symposium: Logical Formalizations of\n",
            "Commonsense Reasoning , volume 46, page 47.\n",
            "Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio\n",
            "Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich K\u00fcttler, Mike Lewis, Wen-tau Yih, Tim Rockt\u00e4schel, et al. 2020. Retrieval-augmented generation\n",
            "for knowledge-intensive nlp tasks. Advances in Neural Information Processing Systems , 33:9459\u20139474.\n",
            "Xiang Lisa Li and Percy Liang. 2021. Pre\ufb01x-Tuning:\n",
            "Optimizing Continuous Prompts for Generation.\n",
            "pages 4582\u20134597.\n",
            "Paul Pu Liang, Chiyu Wu, Louis-Philippe Morency,\n",
            "and Ruslan Salakhutdinov. 2021. Towards understanding and mitigating social biases in language\n",
            "models. In International Conference on Machine\n",
            "Learning , pages 6565\u20136576. PMLR.\n",
            "Opher Lieber, Or Sharir, Barak Lenz, and Yoav\n",
            "Shoham. 2021. Jurassic-1: Technical details and\n",
            "evaluation. Technical report, AI21 Labs.\n",
            "Haochen Liu, Jamell Dacon, Wenqi Fan, Hui Liu, Zitao\n",
            "Liu, and Jiliang Tang. 2019a. Does gender matter?\n",
            "---\n",
            "answering challenge targeting commonsense knowledge. In Proceedings of the 2019 Conference of\n",
            "the North American Chapter of the Association for Computational Linguistics: Human Language\n",
            "Technologies, Volume 1 (Long and Short Papers) , 2019. URL https://aclanthology.\n",
            "org/N19-1421 .\n",
            "Yi Tay, Mostafa Dehghani, Vinh Q. Tran, Xavier Garcia, Jason Wei, Xuezhi Wang, Hyung Won Chung,\n",
            "Dara Bahri, Tal Schuster, Steven Zheng, Denny Zhou, Neil Houlsby, and Donald Metzler. Unifying\n",
            "language learning paradigms, 2022. URL https://arxiv.org/abs/2205.05131 .\n",
            "Romal Thoppilan, Daniel De Freitas, Jamie Hall, Noam Shazeer, Apoorv Kulshreshtha, Heng-Tze\n",
            "Cheng, Alicia Jin, Taylor Bos, Leslie Baker, Yu Du, et al. Lamda: Language models for dialog\n",
            "applications. arXiv preprint arXiv:2201.08239 , 2022. URL https://arxiv.org/abs/\n",
            "2201.08239 .\n",
            "Ashwin Vijayakumar, Michael Cogswell, Ramprasaath Selvaraju, Qing Sun, Stefan Lee, David\n",
            "---\n",
            "Danilo Giampiccolo, Bernardo Magnini, Ido Dagan, and Bill Dolan. The third PASCAL recognizing\n",
            "textual entailment challenge. In Proceedings of the ACL-PASCAL Workshop on Textual Entailment\n",
            "and Paraphrasing , pp. 1\u20139, Prague, June 2007. Association for Computational Linguistics. URL\n",
            "https://www.aclweb.org/anthology/W07-1401 .\n",
            "Aaron Gokaslan and Vanya Cohen. Openwebtext corpus. http://Skylion007.github.io, 2019.\n",
            "Pengcheng He, Xiaodong Liu, Weizhu Chen, and Jianfeng Gao. A hybrid neural network model for\n",
            "commonsense reasoning. arXiv preprint arXiv:1907.11983 , 2019a.\n",
            "Pengcheng He, Yi Mao, Kaushik Chakrabarti, and Weizhu Chen. X-sql: reinforce schema representation with context. arXiv preprint arXiv:1908.08113 , 2019b.\n",
            "Cheng-Zhi Anna Huang, Ashish Vaswani, Jakob Uszkoreit, Ian Simon, Curtis Hawthorne, Noam\n",
            "Shazeer, Andrew M Dai, Matthew D Hoffman, Monica Dinculescu, and Douglas Eck. Music\n",
            "---\n",
            "Generate & rank: A multi-task framework for math word problems. In Findings of the Association\n",
            "for Computational Linguistics: EMNLP 2021 .\n",
            "Alon Talmor, Jonathan Herzig, Nicholas Lourie, and Jonathan Berant. 2019. CommonsenseQA: A\n",
            "question answering challenge targeting commonsense knowledge. NAACL .\n",
            "Alon Talmor, Oyvind Tafjord, Peter Clark, Yoav Goldberg, and Jonathan Berant. 2020. Leap-ofthought: Teaching pre-trained models to systematically reason over implicit knowledge. NeurIPS .\n",
            "Alon Talmor, Ori Yoran, Ronan Le Bras, Chandra Bhagavatula, Yoav Goldberg, Yejin Choi, and\n",
            "Jonathan Berant. 2021. CommonsenseQA 2.0: Exposing the limits of ai through gami\ufb01cation.\n",
            "NeurIPS Track on Datasets and Benchmarks .\n",
            "Yi Tay, Mostafa Dehghani, Vinh Q Tran, Xavier Garcia, Dara Bahri, Tal Schuster, Huaixiu Steven\n",
            "Zheng, Neil Houlsby, and Donald Metzler. 2022. Unifying language learning paradigms. arXiv\n",
            "preprint arXiv:2205.05131 .\n"
          ]
        }
      ],
      "source": [
        "query = \"can you explain why we would want to do rlhf?\"\n",
        "docs = get_docs(query, top_k=25)\n",
        "print(\"\\n---\\n\".join(docs.keys()))"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "lNB4b8sl4bOn"
      },
      "source": [
        "Good, but can we get better?"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ZoFX0vSs4bOn"
      },
      "source": [
        "## Reranking Responses"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "jpHf66Om4bOn"
      },
      "source": [
        "We can easily get the responses we need when we include _many_ responses, but this doesn't work well with LLMs. The recall performance for LLMs [decreases as we add more into the context window](https://www.pinecone.io/blog/why-use-retrieval-instead-of-larger-context/) \u2014 we call this excessive filling of the context window _\"context stuffing\"_.\n",
        "\n",
        "Fortunately reranking offers us a solution that helps us find those records that may not be within the top-3 results, and pull them into a smaller set of results to be given to the LLM.\n",
        "\n",
        "We will use Cohere's rerank endpoint for this, to use it you will need a [Cohere API key](https://dashboard.cohere.com/api-keys). Once you have your key you use it to create authenticate your Cohere client like so:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 14,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "VfiZ-KUV4bOo",
        "outputId": "d564a7fe-0e5d-46e8-bc44-cb3855e07a76"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "\u00b7\u00b7\u00b7\u00b7\u00b7\u00b7\u00b7\u00b7\u00b7\u00b7\n"
          ]
        }
      ],
      "source": [
        "import cohere\n",
        "\n",
        "os.environ[\"COHERE_API_KEY\"] = os.getenv(\"COHERE_API_KEY\") or getpass.getpass()\n",
        "# init client\n",
        "co = cohere.Client(os.environ[\"COHERE_API_KEY\"])"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Xxt_cNPI4bOo"
      },
      "source": [
        "Now we can rerank our results with `co.rerank`. Let's try it with our earlier results."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 15,
      "metadata": {
        "id": "Q_ucNH2dIXKD"
      },
      "outputs": [],
      "source": [
        "rerank_docs = co.rerank(\n",
        "    query=query, documents=docs.keys(), top_n=25, model=\"rerank-english-v2.0\"\n",
        ")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "KeW4znDwJJjj"
      },
      "source": [
        "This returns a list of `RerankResult` objects:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 16,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "z9auAxaTJEPU",
        "outputId": "da8d4507-4f86-4a44-abbe-eeb6810b7f0a"
      },
      "outputs": [
        {
          "data": {
            "text/plain": [
              "cohere.responses.rerank.RerankResult"
            ]
          },
          "execution_count": 16,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "type(rerank_docs[0])"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "aMWXA4YbJf-U"
      },
      "source": [
        "We access the text content of the docs like so:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 17,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 143
        },
        "id": "4ukXGwJ4JQhh",
        "outputId": "ee5262fd-9b4b-402a-c95a-dc6d53a5703a"
      },
      "outputs": [
        {
          "data": {
            "application/vnd.google.colaboratory.intrinsic+json": {
              "type": "string"
            },
            "text/plain": [
              "'significant for 3 out of 7 datasets.\\n28\\nK New Tasks\\nConstrained Generation We introduce \u201cCommonGen-Hard,\" a more challenging extension of the\\nCommonGen dataset (Lin et al., 2020), designed to test state-of-the-art language models\u2019 advanced\\ncommonsense reasoning, contextual understanding, and creative problem-solving. CommonGenHard requires models to generate coherent sentences incorporating 20-30 concepts, rather than only\\nthe 3-5 related concepts given in CommonGen. SELF-REFINE focuses on iterative creation with\\nintrospective feedback, making it suitable for evaluating the effectiveness of language models on the\\nCommonGen-Hard task.\\nAcronym Generation Acronym generation requires an iterative refinement process to create\\nconcise and memorable representations of complex terms or phrases, involving tradeoffs between\\nlength, ease of pronunciation, and relevance, and thus serves as a natural testbed for our approach.\\nWe source a dataset of 250 acronyms4and manually prune it to remove offensive or uninformative\\nacronyms.\\nL Code Readability\\nOrthogonal to the correctness, readability is another important quality of a piece of code: though not\\nrelated to the execution results of the code, code readability may significantly affect the usability,\\nupgradability, and ease of maintenance of an entire codebase. In this section, we consider the problem'"
            ]
          },
          "execution_count": 17,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "rerank_docs[0].document[\"text\"]"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "bOUw6AVFDhG9"
      },
      "source": [
        "The reordered results look like so:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 18,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "_-lOqKxhIy4C",
        "outputId": "4f7cf0ff-9a76-461f-d1cd-d2d1730390ab"
      },
      "outputs": [
        {
          "data": {
            "text/plain": [
              "[20,\n",
              " 22,\n",
              " 6,\n",
              " 15,\n",
              " 13,\n",
              " 4,\n",
              " 19,\n",
              " 1,\n",
              " 12,\n",
              " 23,\n",
              " 24,\n",
              " 17,\n",
              " 16,\n",
              " 18,\n",
              " 21,\n",
              " 7,\n",
              " 0,\n",
              " 11,\n",
              " 8,\n",
              " 14,\n",
              " 9,\n",
              " 5,\n",
              " 2,\n",
              " 3,\n",
              " 10]"
            ]
          },
          "execution_count": 18,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "[docs[doc.document[\"text\"]] for doc in rerank_docs]"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "SKiUwIikMGU1"
      },
      "source": [
        "Let's write a function to allow us to more easily compare the original results vs. reranked results."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 19,
      "metadata": {
        "id": "TfFFNLu2MLrt"
      },
      "outputs": [],
      "source": [
        "def compare(query: str, top_k: int, top_n: int):\n",
        "    # first get vec search results\n",
        "    docs = get_docs(query, top_k=top_k)\n",
        "    i2doc = {docs[doc]: doc for doc in docs.keys()}\n",
        "    # rerank\n",
        "    rerank_docs = co.rerank(\n",
        "        query=query, documents=docs.keys(), top_n=top_n, model=\"rerank-english-v2.0\"\n",
        "    )\n",
        "    original_docs = []\n",
        "    reranked_docs = []\n",
        "    # compare order change\n",
        "    for i, doc in enumerate(rerank_docs):\n",
        "        rerank_i = docs[doc.document[\"text\"]]\n",
        "        print(str(i)+\"\\t->\\t\"+str(rerank_i))\n",
        "        if i != rerank_i:\n",
        "            reranked_docs.append(f\"[{rerank_i}]\\n\"+doc.document[\"text\"])\n",
        "            original_docs.append(f\"[{i}]\\n\"+i2doc[i])\n",
        "    for orig, rerank in zip(original_docs, reranked_docs):\n",
        "        print(\"ORIGINAL:\\n\"+orig+\"\\n\\nRERANKED:\\n\"+rerank+\"\\n\\n---\\n\")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "2WZim2vZDhG9"
      },
      "source": [
        "Beginning with our `\"can you explain why we would want to do rlhf?\"` query, let's take a look at the top-3 results with / without reranking:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 20,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "kwcRIVX-Ng6N",
        "outputId": "6d9d6104-148e-456b-8ef5-71f718f660f8"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "0\t->\t20\n",
            "1\t->\t22\n",
            "2\t->\t6\n",
            "ORIGINAL:\n",
            "[0]\n",
            "Bill Yuchen Lin, Wangchunshu Zhou, Ming Shen, Pei Zhou, Chandra Bhagavatula, Yejin Choi,\n",
            "and Xiang Ren. CommonGen: A constrained text generation challenge for generative commonsense reasoning. In Findings of the Association for Computational Linguistics: EMNLP\n",
            "2020 , pp. 1823\u20131840, Online, November 2020. Association for Computational Linguistics.\n",
            "doi: 10.18653/v1/2020.\ufb01ndings-emnlp.165. URL https://aclanthology.org/2020.\n",
            "findings-emnlp.165 .\n",
            "Alisa Liu, Maarten Sap, Ximing Lu, Swabha Swayamdipta, Chandra Bhagavatula, Noah A. Smith,\n",
            "and Yejin Choi. DExperts: Decoding-time controlled text generation with experts and antiexperts. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume\n",
            "1: Long Papers) , pp. 6691\u20136706, Online, August 2021. Association for Computational Linguistics. doi: 10.18653/v1/2021.acl-long.522. URL https://aclanthology.org/2021.\n",
            "acl-long.522 .\n",
            "\n",
            "RERANKED:\n",
            "[20]\n",
            "significant for 3 out of 7 datasets.\n",
            "28\n",
            "K New Tasks\n",
            "Constrained Generation We introduce \u201cCommonGen-Hard,\" a more challenging extension of the\n",
            "CommonGen dataset (Lin et al., 2020), designed to test state-of-the-art language models\u2019 advanced\n",
            "commonsense reasoning, contextual understanding, and creative problem-solving. CommonGenHard requires models to generate coherent sentences incorporating 20-30 concepts, rather than only\n",
            "the 3-5 related concepts given in CommonGen. SELF-REFINE focuses on iterative creation with\n",
            "introspective feedback, making it suitable for evaluating the effectiveness of language models on the\n",
            "CommonGen-Hard task.\n",
            "Acronym Generation Acronym generation requires an iterative refinement process to create\n",
            "concise and memorable representations of complex terms or phrases, involving tradeoffs between\n",
            "length, ease of pronunciation, and relevance, and thus serves as a natural testbed for our approach.\n",
            "We source a dataset of 250 acronyms4and manually prune it to remove offensive or uninformative\n",
            "acronyms.\n",
            "L Code Readability\n",
            "Orthogonal to the correctness, readability is another important quality of a piece of code: though not\n",
            "related to the execution results of the code, code readability may significantly affect the usability,\n",
            "upgradability, and ease of maintenance of an entire codebase. In this section, we consider the problem\n",
            "\n",
            "---\n",
            "\n",
            "ORIGINAL:\n",
            "[1]\n",
            "2017. Asian Federation of Natural Language Processing. URL https://aclanthology.\n",
            "org/I17-1099 .\n",
            "Bill Yuchen Lin, Wangchunshu Zhou, Ming Shen, Pei Zhou, Chandra Bhagavatula, Yejin Choi, and\n",
            "Xiang Ren. CommonGen: A constrained text generation challenge for generative commonsense\n",
            "reasoning. In Findings of the Association for Computational Linguistics: EMNLP 2020 , pp.\n",
            "1823\u20131840, Online, 2020. Association for Computational Linguistics. doi: 10.18653/v1/2020.\n",
            "\ufb01ndings-emnlp.165.\n",
            "Chin-Yew Lin. Rouge: A package for automatic evaluation of summaries. In Text summarization\n",
            "branches out , pp. 74\u201381, 2004.\n",
            "Siqi Liu, Zhenhai Zhu, Ning Ye, Sergio Guadarrama, and Kevin Murphy. Improved image captioning\n",
            "via policy gradient optimization of spider. In Proceedings of the IEEE international conference on\n",
            "computer vision , pp. 873\u2013881, 2017.\n",
            "Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike\n",
            "\n",
            "RERANKED:\n",
            "[22]\n",
            "answering challenge targeting commonsense knowledge. In Proceedings of the 2019 Conference of\n",
            "the North American Chapter of the Association for Computational Linguistics: Human Language\n",
            "Technologies, Volume 1 (Long and Short Papers) , 2019. URL https://aclanthology.\n",
            "org/N19-1421 .\n",
            "Yi Tay, Mostafa Dehghani, Vinh Q. Tran, Xavier Garcia, Jason Wei, Xuezhi Wang, Hyung Won Chung,\n",
            "Dara Bahri, Tal Schuster, Steven Zheng, Denny Zhou, Neil Houlsby, and Donald Metzler. Unifying\n",
            "language learning paradigms, 2022. URL https://arxiv.org/abs/2205.05131 .\n",
            "Romal Thoppilan, Daniel De Freitas, Jamie Hall, Noam Shazeer, Apoorv Kulshreshtha, Heng-Tze\n",
            "Cheng, Alicia Jin, Taylor Bos, Leslie Baker, Yu Du, et al. Lamda: Language models for dialog\n",
            "applications. arXiv preprint arXiv:2201.08239 , 2022. URL https://arxiv.org/abs/\n",
            "2201.08239 .\n",
            "Ashwin Vijayakumar, Michael Cogswell, Ramprasaath Selvaraju, Qing Sun, Stefan Lee, David\n",
            "\n",
            "---\n",
            "\n",
            "ORIGINAL:\n",
            "[2]\n",
            "forComputational Linguistics: ACL-IJCNLP 2021,\n",
            "pages 596\u2013610, Online. Association for Computational Linguistics.\n",
            "Bill Yuchen Lin, Wangchunshu Zhou, Ming Shen,\n",
            "Pei Zhou, Chandra Bhagavatula, Yejin Choi, and\n",
            "Xiang Ren. 2020. CommonGen: A constrained\n",
            "text generation challenge for generative commonsense reasoning. In Findings oftheAssociation\n",
            "forComputational Linguistics: EMNLP 2020, pages\n",
            "1823\u20131840, Online. Association for Computational\n",
            "Linguistics.\n",
            "Chin-Yew Lin. 2004. Rouge: A package for automatic\n",
            "evaluation of summaries. In Text summarization\n",
            "branches out, pages 74\u201381.\n",
            "Wang Ling, Dani Yogatama, Chris Dyer, and Phil Blunsom. 2017. Program induction by rationale generation: Learning to solve and explain algebraic\n",
            "word problems. In Proceedings ofthe55th Annual\n",
            "Meeting ofthe Association for Computational\n",
            "Linguistics (V olume 1:Long Papers), pages 158\u2013\n",
            "167, Vancouver, Canada. Association for Computational Linguistics.\n",
            "Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis,\n",
            "\n",
            "RERANKED:\n",
            "[6]\n",
            "from large language models make small reasoners\n",
            "better. ArXiv preprint , abs/2210.06726.\n",
            "Yifei Li, Zeqi Lin, Shizhuo Zhang, Qiang Fu, Bei Chen,\n",
            "Jian-Guang Lou, and Weizhu Chen. 2022b. On the\n",
            "advance of making language models better reasoners.\n",
            "ArXiv preprint , abs/2206.02336.\n",
            "Bill Yuchen Lin, Wangchunshu Zhou, Ming Shen, Pei\n",
            "Zhou, Chandra Bhagavatula, Yejin Choi, and Xiang\n",
            "Ren. 2020. CommonGen: A constrained text generation challenge for generative commonsense reasoning. In Findings of the Association for Computational Linguistics: EMNLP 2020 , pages 1823\u20131840,\n",
            "Online. Association for Computational Linguistics.\n",
            "Wang Ling, Dani Yogatama, Chris Dyer, and Phil Blunsom. 2017. Program induction by rationale generation: Learning to solve and explain algebraic word\n",
            "problems. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics\n",
            "(Volume 1: Long Papers) , pages 158\u2013167, Vancouver,\n",
            "Canada. Association for Computational Linguistics.\n",
            "Chenzhengyi Liu, Jie Huang, Kerui Zhu, and Kevin\n",
            "\n",
            "---\n",
            "\n"
          ]
        }
      ],
      "source": [
        "compare(query, 25, 3)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "M-WFVvluDhG-"
      },
      "source": [
        "Both results from reranking provide many more reasons as to why we would want to use RLHF than the original records. Let's try another query:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 21,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "WtqdxP9cQMUP",
        "outputId": "57487635-45df-4896-f249-973db16b9890"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "0\t->\t11\n",
            "1\t->\t22\n",
            "2\t->\t17\n",
            "ORIGINAL:\n",
            "[0]\n",
            "Bill Yuchen Lin, Wangchunshu Zhou, Ming Shen, Pei Zhou, Chandra Bhagavatula, Yejin Choi,\n",
            "and Xiang Ren. CommonGen: A constrained text generation challenge for generative commonsense reasoning. In Findings of the Association for Computational Linguistics: EMNLP\n",
            "2020 , pp. 1823\u20131840, Online, November 2020. Association for Computational Linguistics.\n",
            "doi: 10.18653/v1/2020.\ufb01ndings-emnlp.165. URL https://aclanthology.org/2020.\n",
            "findings-emnlp.165 .\n",
            "Alisa Liu, Maarten Sap, Ximing Lu, Swabha Swayamdipta, Chandra Bhagavatula, Noah A. Smith,\n",
            "and Yejin Choi. DExperts: Decoding-time controlled text generation with experts and antiexperts. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume\n",
            "1: Long Papers) , pp. 6691\u20136706, Online, August 2021. Association for Computational Linguistics. doi: 10.18653/v1/2021.acl-long.522. URL https://aclanthology.org/2021.\n",
            "acl-long.522 .\n",
            "\n",
            "RERANKED:\n",
            "[11]\n",
            "movie, and this film is one of those films where he can be convincing in it (and his\n",
            "trademark acting, as you can see in the\n",
            "27\n",
            "Published as a conference paper at ICLR 2023\n",
            "B.4 C OMMON GEN\n",
            "B.4.1 S ETUP\n",
            "CommonGen (Lin et al., 2020) deals with task of generating coherent sentences describing an\n",
            "input set of concepts (eg. \"a man is throwing a frisbee\"). For training RL methods, we consider\n",
            "3 traditional lexical rewards namely Rouge-1, Rouge-avg (which is an average of Rouge-1, 2 and\n",
            "L) and meteor. Additionally, we also train with task-speci\ufb01c rewards such as CIDEr (Vedantam\n",
            "et al., 2015), SPICE (Anderson et al., 2016) and SPiDer (Liu et al., 2017) which is a just a linear\n",
            "combination of both with equal weights. We chose T5-base as the base LM since it is well-suited\n",
            "for structure to text tasks. We additionally note that concept set inputs are pre\ufb01xed with \"generate a\n",
            "sentence with:\" to encourage exploration.\n",
            "During our initial experiments when \ufb01ne-tuning directly on LM, we observed that policy learns to\n",
            "repeat the prompted concepts in order to maximize rewards resulting in a well-known problem of\n",
            "\n",
            "---\n",
            "\n",
            "ORIGINAL:\n",
            "[1]\n",
            "2017. Asian Federation of Natural Language Processing. URL https://aclanthology.\n",
            "org/I17-1099 .\n",
            "Bill Yuchen Lin, Wangchunshu Zhou, Ming Shen, Pei Zhou, Chandra Bhagavatula, Yejin Choi, and\n",
            "Xiang Ren. CommonGen: A constrained text generation challenge for generative commonsense\n",
            "reasoning. In Findings of the Association for Computational Linguistics: EMNLP 2020 , pp.\n",
            "1823\u20131840, Online, 2020. Association for Computational Linguistics. doi: 10.18653/v1/2020.\n",
            "\ufb01ndings-emnlp.165.\n",
            "Chin-Yew Lin. Rouge: A package for automatic evaluation of summaries. In Text summarization\n",
            "branches out , pp. 74\u201381, 2004.\n",
            "Siqi Liu, Zhenhai Zhu, Ning Ye, Sergio Guadarrama, and Kevin Murphy. Improved image captioning\n",
            "via policy gradient optimization of spider. In Proceedings of the IEEE international conference on\n",
            "computer vision , pp. 873\u2013881, 2017.\n",
            "Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike\n",
            "\n",
            "RERANKED:\n",
            "[22]\n",
            "answering challenge targeting commonsense knowledge. In Proceedings of the 2019 Conference of\n",
            "the North American Chapter of the Association for Computational Linguistics: Human Language\n",
            "Technologies, Volume 1 (Long and Short Papers) , 2019. URL https://aclanthology.\n",
            "org/N19-1421 .\n",
            "Yi Tay, Mostafa Dehghani, Vinh Q. Tran, Xavier Garcia, Jason Wei, Xuezhi Wang, Hyung Won Chung,\n",
            "Dara Bahri, Tal Schuster, Steven Zheng, Denny Zhou, Neil Houlsby, and Donald Metzler. Unifying\n",
            "language learning paradigms, 2022. URL https://arxiv.org/abs/2205.05131 .\n",
            "Romal Thoppilan, Daniel De Freitas, Jamie Hall, Noam Shazeer, Apoorv Kulshreshtha, Heng-Tze\n",
            "Cheng, Alicia Jin, Taylor Bos, Leslie Baker, Yu Du, et al. Lamda: Language models for dialog\n",
            "applications. arXiv preprint arXiv:2201.08239 , 2022. URL https://arxiv.org/abs/\n",
            "2201.08239 .\n",
            "Ashwin Vijayakumar, Michael Cogswell, Ramprasaath Selvaraju, Qing Sun, Stefan Lee, David\n",
            "\n",
            "---\n",
            "\n",
            "ORIGINAL:\n",
            "[2]\n",
            "forComputational Linguistics: ACL-IJCNLP 2021,\n",
            "pages 596\u2013610, Online. Association for Computational Linguistics.\n",
            "Bill Yuchen Lin, Wangchunshu Zhou, Ming Shen,\n",
            "Pei Zhou, Chandra Bhagavatula, Yejin Choi, and\n",
            "Xiang Ren. 2020. CommonGen: A constrained\n",
            "text generation challenge for generative commonsense reasoning. In Findings oftheAssociation\n",
            "forComputational Linguistics: EMNLP 2020, pages\n",
            "1823\u20131840, Online. Association for Computational\n",
            "Linguistics.\n",
            "Chin-Yew Lin. 2004. Rouge: A package for automatic\n",
            "evaluation of summaries. In Text summarization\n",
            "branches out, pages 74\u201381.\n",
            "Wang Ling, Dani Yogatama, Chris Dyer, and Phil Blunsom. 2017. Program induction by rationale generation: Learning to solve and explain algebraic\n",
            "word problems. In Proceedings ofthe55th Annual\n",
            "Meeting ofthe Association for Computational\n",
            "Linguistics (V olume 1:Long Papers), pages 158\u2013\n",
            "167, Vancouver, Canada. Association for Computational Linguistics.\n",
            "Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis,\n",
            "\n",
            "RERANKED:\n",
            "[17]\n",
            "the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on\n",
            "Natural Language Processing (Volume 1: Long Papers) . Association for Computational Linguistics, Online, 4582\u20134597.\n",
            "https://doi.org/10.18653/v1/2021.acl-long.353\n",
            "[72] Xiang Lisa Li, John Thickstun, Ishaan Gulrajani, Percy Liang, and Tatsunori B. Hashimoto. 2022. Diffusion-LM\n",
            "Improves Controllable Text Generation. https://doi.org/10.48550/ARXIV.2205.14217\n",
            "[73] Alisa Liu, Maarten Sap, Ximing Lu, Swabha Swayamdipta, Chandra Bhagavatula, Noah A. Smith, and Yejin Choi.\n",
            "2021. DExperts: Decoding-Time Controlled Text Generation with Experts and Anti-Experts. In Proceedings of the\n",
            "59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on\n",
            "Natural Language Processing (Volume 1: Long Papers) . Association for Computational Linguistics, Online, 6691\u20136706.\n",
            "https://doi.org/10.18653/v1/2021.acl-long.522\n",
            "\n",
            "---\n",
            "\n"
          ]
        }
      ],
      "source": [
        "compare(\"what is red teaming?\", top_k=25, top_n=3)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "BHxkHsXKDhHC"
      },
      "source": [
        "Again, the results provide more relevant responses when using reranking rather than the original search."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "u8OfJFwq4bOo"
      },
      "source": [
        "Don't forget to delete your index when you're done to save resources!"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 22,
      "metadata": {
        "id": "LQiU0IDl4bOo"
      },
      "outputs": [],
      "source": [
        "pc.delete_index(index_name)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "0gThAy0k4bOo"
      },
      "source": [
        "---"
      ]
    }
  ],
  "metadata": {
    "colab": {
      "provenance": []
    },
    "kernelspec": {
      "display_name": "ml",
      "language": "python",
      "name": "python3"
    },
    "language_info": {
      "codemirror_mode": {
        "name": "ipython",
        "version": 3
      },
      "file_extension": ".py",
      "mimetype": "text/x-python",
      "name": "python",
      "nbconvert_exporter": "python",
      "pygments_lexer": "ipython3",
      "version": "3.9.12"
    },
    "orig_nbformat": 4
  },
  "nbformat": 4,
  "nbformat_minor": 0
}